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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the 
fee set forth in 37 CFR 1 .1 7(e), was filed in this application after final rejection. Since 
this application is eligible for continued examination under 37 CFR 1.114, and the fee 
set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office 
action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
October 12 th , 2005 has been entered. 

Response to Amendment 

2. Applicant's after-final amendment filed August 17 th , 2005 has now been 
entered in view of the RCE cited above. 

3. Applicant has amended Claims 1,12 and 1 8. No new claims have been 
added. Claims 1-29 remain pending. 

4. Applicant's remarks with regard to currently amended claims are not found 
fully persuasive for at least the following reasons: 

5. Applicant has amended the independent claims 1,12 and 18 to include 
the limitations of: 
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"determining 3-D depth of the plurality of pixels in the base image by 
simultaneously matching correspondence to a plurality of pixels in three or more 
images / 7 . 

It is unclear how "matching correspondence to a plurality of pixels in three or 
more images " can be done simultaneously . 

Applicant claims that support for these claimed features can be found in the 
specification in Fig. 3; page 14, lines 10-15; and page 15, lines 17-19. 

Fig. 3 illustrates a multi-plane stereo arrangement of cameras. There are six 
cameras or view points which meets the limitation of the feature: " three or more 
images ". Examiner points out that Chen also teaches this feature and even uses the 
same number of six stereo images (page 34, top left Example 4). 

As best gathered from the specification, the present invention performs 3-D 
image determination by using multi-pane stereo imaging which functions by determining 
correspondence between pairs of images using only two images at a time and 
measuring correspondence between those two images. The phrase of "matching 
correspondence" with a plurality of pixels, as best interpreted can only be done between 
two images at a time or two sets of data at a time. Differences between image pairs are 
determined as shown in the specification on page 14, lines 5-15 namely the equation 
determining Di. Then a "matching cost E" is determined using the differences of 
multiple images using a summation of the differences between the initial image and 
each subsequent image. Applicant points to this passage as providing support for the 
simultaneous matching correspondence of a plurality of pixels in a base image and 
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three or more images representing other image views. However, the calculation of a 
matching cost E is not the same as "determining 3-D depth of the plurality of pixels in 
the base image by simultaneously matching correspondence to a plurality of pixels in 
three or more images .". Even the summation disclosed determines the similarity or 
difference between the initial image and only one of the plurality of other images in each 
iteration. So the correspondence between the plurality of images and the initial images 
is not completely simultaneous. 

Applicant further points to page 1 5, lines 1 7-1 9 to provide support for the added 
limitations. This passage reads as follows: "Without an initial value for g, one has to 
resort to a global search on the ray cO+gL(ui,vj). Now, if it happens that one of (uj,vj)'s 
four neighbors has a known depth parameter g\ then based on object surface 
cohesivity, it is likely that gj=g\" This passage seems to be cited to support the 
calculation of the depth values (g) for the pixels, but it is still unclear how the depth 
values are solved for as claimed "by simultaneously matching correspondence to a 
plurality of pixels in three or more images .". 

Therefore in the further treatment of the claims, simultaneously is given little 
weight. As best interpreted, the reference of Chen still reads on the presently claimed 
invention because Chen clearly uses three or more images (page 34, top left Example 
4). 

It is respectfully requested that Applicant explain or clarify how "matching 
correspondence to a plurality of pixels in three or more images" is performed 
simultaneously in the present invention. 
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U.S. Patent 6,834,1 19 also owned by the present assignee is repeatedly 
mentioned in the present application as being incorporated by reference. This patent 
also seems to teach finding matching through image pairs in an interation method 
similar to that described above as well. 

Applicant further argues that Chen does not disclose the previously presented 
limitation of the claim of: 

"tracing pixels in a virtual piecewise continuous depth surface by spatial 
propagation starting from the detected pixels in the base image by using the matching 
and corresponding plurality of pixels in the three or more images to create the virtual 
piecewise continuous depth surface viewed from the base image, each successfully 
traced pixel being associated with a depth in the scene viewed from the base image." 

Examiner points out that the passage previously cited still reads on the limitation 
as discussed in the previous office action on page 6. This citation is provided again 
below for convenience: 

Steps (1.a.)-(1.b) are typical of any stereo matching applications. The "matching 
correspondence" is discussed in Section 3. 1 of [Chen, VSMM99J. The cross-correlation 
Qu,v,dis indicative of the matching correspondence between pixels of different images, 
each of which are representative of disparate views of the scene. 

( 1.c.) Tracing pixels (i.e. surface tracing) in a virtual piecewise continuous 
depth surface (i.e. disparity surface d(u. v)) by spatial propagation 
([Chen, VSMM99] section 3.2, paragraph 1, sentence 4) starting 
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from the detected pixels in the base image (i.e. all pixels (u,v) of the 
base image) by using the matching and corresponding plurality of 
pixels in the plurality of images (e.g. seed pixels that satisfy a given 
cross-correlation criterion - [Chen, VSMM99] page 31, right 
column, paragraph 1, sentences 1-2) to create the virtual piecewise 
continuous depth surface viewed from the base image. Please refer 
to Section 3 of [Chen, VSMM99J. 
(1.d.) Each successfully traced pixel is associated with a depth in the 
scene viewed from the base image. This follows from the fact that 
traced pixels lie on the disparity surface. Again, the disparity 
surface is indicative of the depth of all visible surfaces of a given 
scene (page 29, right column, last paragraph, sentences 1-3). 

The rest of Applicant's arguments seem to hinge on the argument presented in 
view of the independent claims, that the presently claimed invention uses a plurality of 
image correspondences simultaneously and that the reference of Chen is only using 
two images at a time. In view of the above presented discussion the rejection of the 
claims 1, 12 and 18 is maintained, as well as the rejection of the dependent claims. 



Claim Rejections • 35 USC §112 

The following is a quotation of the first paragraph of 35 U.S.C. 112: 
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The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention, 

6. Claims 1,12 and 18 are rejected under 35 U.S.C. 112, first paragraph, as 
failing to comply with the written description requirement. The claim(s) contains subject 
matter which was not described in the specification in such a way as to reasonably 
convey to one skilled in the relevant art that the inventor(s), at the time the application 
was filed, had possession of the claimed invention. Applicant has amended 
independent claims to include the limitation of: 

"determining 3-D depth of the plurality of pixels in the base image by 
simultaneously matching correspondence to a plurality of pixels in three or more 
images . Refer to the discussion presented above in response to applicants remarks. It 
is unclear how a matching correspondence can be performed simultaneously between a 
bas image and multiple other image and further it is unclear how the 3-D depth 
information is determined by matching correspondence. It is respectfully request that 
Applicant explain and clarify how this is possible. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 

form the basis for the rejections under this section made in this Office action: 

(b) the invention was patented or described in a printed publication in this or a 
foreign country or in public use or on sale in this country, more than one year 
prior to the date of application for patent in the United States. 
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7. Claims 1, 4-9, 12-18, and 21-26 are rejected under 35 U.S.C. 102(b) as 
being anticipated by [Chen, VSMM99] (Chen and Medioni , "A Volumetric Stereo 
Matching Method: Application to Image-Based Modeling", IEEE 1999). 

The following is in regard to Claim 1. [Chen, VSMM99] describes a volumetric 
stereo matching method. The method involves the recovery of depth information - 
namely, the disparity surface d(u,v) (see [Chen, VSMM99]: Abstract, sentence 1; page 
29, right column, last paragraph 1 , sentences 1-3; and page 30, left column, paragraph 
1 , sentences 2-3) - for pixels of a base image representing a view of a scene. The 
method comprises at least the following steps: 

(1 .a.) Detecting a plurality of pixels in a base image that represents a first 
view of a scene wherein the first view is one of at least three views 
of a scene (e.g. capturing images from various viewpoints of the 
scene). It should be apparent from the various stereo images 
shown in [Chen, VSMM99] that the volumetric stereo matching 
method involves this step. Any one of these images can be 
arbitrarily denoted as a "base image". See Also Example 4 on 
page 34, where 6 views of a teapot are used. 
(1 .b.) Determining 3-D depth of the plurality of pixels in the base image 



i 



When referring to paragraphs in the cited references, the convention followed here is that the paragraph number is assigned to 
paragraphs of a given column (if applicable) or section, sequentially, beginning with the first fiilt paragraph. Paragraphs that 
carry over to other columns will be referred to as the last paragraph of the column in which they began. 
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by simultaneously matching correspondence to a plurality of pixels 
in three or more images each image representing one of the at 
least three views of the scene wherein each of the at least three 
views of the scene are situated in a non-linear arrangement and 
are further oriented in a plurality of non-parallel planes relative to 
each other. See Also Example 4 on page 34, where 6 views of a 
teapot are used. Chen discloses using views of a teapot as it is 
rotated, the views of which would therefore not lie on parallel 
planes. 

Steps (1.a.)-(1.b) are typical of any stereo matching applications. The "matching 
correspondence" is discussed in Section 3.1 of [Chen, VSMM99]. The cross-correlation 
Quxd'is indicative of the matching correspondence between pixels of different images, 
each of which are representative of disparate views of the scene. 

(1 .c.) Tracing pixels (i.e. surface tracing) in a virtual piecewise continuous 
depth surface (i.e. disparity surface d{u.v)) by spatial propagation 
([Chen, VSMM99] section 3.2, paragraph 1, sentence 4) starting 
from the detected pixels in the base image (i.e. all pixels (u,v) of the 
base image) by using the matching and corresponding plurality of 
pixels in the three or more images (e.g. seed pixels that satisfy a 
given cross-correlation criterion - [Chen, VSMM99] page 31, right 
column, paragraph 1, sentences 1-2) to create the virtual piecewise 
continuous depth surface viewed from the base image. Please refer 
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to Section 3 of [Chen, VSMM99]. 
(1 .d.) Each successfully traced pixel is associated with a depth in the 
scene viewed from the base image. This follows from the fact that 
traced pixels lie on the disparity surface. Again, the disparity 
surface is indicative of the depth of all visible surfaces of a given 
scene (page 29, right column, last paragraph, sentences 1-3). 

In this manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms 

sufficiently to the method proposed by the Applicant in claim 1 . 

The following is in regard to Claim 18. Claim 1 8 recites essentially the same 

limitations as claim 1 . (The 3D reconstruction instructions merely implement the depth 

recovery method of Claim 1). Therefore, with regard to claim 18, remarks analogous to 

those presented above relative to claim 1 are applicable. 

The following is in regard to Claim 4. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that adequately conforms to the depth 
recovery method of Claim 1 . The volumetric stereo matching method of [Chen, 
VSMM99] further involves: 

Propagating a front of a virtual piece of a continuous depth surface 
(e.g. a surface front- this terminology is used in sentence 5 of the 
last paragraph of the right column on page 31) to at least one 
neighboring pixel (e.g. each 4-neighboroi (u,v) - Fig. 4 of [Chen, 
VSMM99]) starting from the detected pixels (e.g. pixel (u,v)) in the 
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base image. 

It should be apparent from the algorithm listed in Fig. 4 of [Chen, VSMM99] that the 
boundary (front) of the disparity surface is propagated. In this manner, the volumetric 
stereo matching method of [Chen, VSMM99] conforms sufficiently to the method 
proposed by the Applicant in Claim 4. 

The following is in regard to Claim 21. Claim 21 recites essentially the same 
limitations as claim 4. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 4). Therefore, with regard to claim 21, remarks analogous to 
those presented above relative to claim 4 are applicable. 

The following is in regard to Claims 5-6. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that adequately conforms to the depth 
recovery method of Claim 4. Tracing the disparity surface according to the volumetric 
stereo matching method of [Chen, VSMM99] further involves the following: 

(5.a.) Determining when a boundary is reached between two propagating 
fronts of virtual pieces of a continuous depth surface ([Chen, 
VSMM99] page 31, right column, last paragraph, sentence 5). 
If the boundaries of two propagating fronts coincide then: 

(6. a.) Comparing the cross-correlations of the two propagating fronts 
about the reached boundary. This comparison is performed, for 
example, in the following step shown in [Chen, VSMM99] Fig. 4: 
disparity(u>') = Pu>',cTpp u'y,d'/nd' pcT 
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Clearly, a comparison of the cross-correlations, nu',v',d'ar\dn 
u',v',d'[ is tantamount to a comparison of the matching costs of the 
two surface fronts 
(6.b.) The surface front with the greatest cross-correlation prevails 

([Chen, VSMM99] page 31, right column, last paragraph, sentence 
5). In other words, the propagation of the front with the lower cross- 
correlation is stopped. 
Step (6.b.) is equivalent to stopping the propagation of the front with the higher 
compared matching cost 2 . In this manner, the a volumetric stereo matching method of 
[Chen, VSMM99] conforms sufficiently to both the methods of Claim 5 and the method 
of Claim 6. 

The following is in regard to Claim 22-23. Claims 22-23 recite essentially the 
same limitations as claim 5-6, respectively. (The 3D reconstruction instructions of these 
claims merely implement the depth recovery methods of Claim 5-6). Therefore, with 
regard to claim 22-23, remarks analogous to those presented above relative to claim 5- 
6 are respectively applicable. 

The following is in regard to Claim 7. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that adequately conforms to the depth 

2 To see this recall how the Applicant defines matching cost (page 9, lines 1 0- 1 5 of the Applicant's disclosure and, in 
particular, equation (8) on page 17). The matching cost is low for pixels that are likely matches and is high for unlikely 
matches. The relationship between the matching cost and cross-correlation (NCC) is shown in the Applicant's equation (8). 
Clearly, a high NCC results in a low matching cost E(g) and indicates a likely match, whereas a low NCC results in a high 
matching cost and indicates an unlikely match. Therefore, by stopping the propagation of the front with the lower NCC, one 
has effectively stopped the propagation of the front with the higher matching cost 
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recovery method of Claim 1. Tracing the disparity surface according to the volumetric 

stereo matching method of [Chen, VSMM99] further involves the following: 
Propagating a front of a virtual piece of a continuous depth surface 
to at least one neighboring pixel surrounded by a predefined size 
window in the continuous depth surface. According to the 
volumetric stereo matching method of [Chen, VSMM99], the 
normalized cross-correlation (NCC) over a window (e.g. one with 
dimensions and centered at pixels {u,v) and (u+d,v)) is used as the 
similarity measurement. See Section 3.1 of [Chen, VSMM99]. 

In this manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms 

sufficiently to the method of Claim 7. 

The following is in regard to Claim 24. Claim 24 recites essentially the same 

limitations as claim 7. (The 3D reconstruction instructions merely implement the depth 

recovery method of Claim 7). Therefore, with regard to claim 24, remarks analogous to 

those presented above relative to claim 7 are applicable. 

The following is in regard to Claim 8-9 and 25-26. As shown above, [Chen, 
VSMM99] discloses a volumetric stereo matching method that adequately satisfies the 
limitations of Claims 7 and 24. The limitations set forth in Claims 8-9 and 25-26 were 
addressed previously in the discussion above relating to Claims 5-6 and 22-23. The 
details will not be repeated here. 
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The following is in regard to Claim 12. [Chen, VSMM99] discloses a volumetric 
stereo matching method that includes the following: 

(12. a.) Providing a plurality of seed pixels that represent 3-D depth of the 
plurality of pixels in the base image view of a scene (i.e. seed 
voxels - [Chen, VSMM99] page 31 , right column, paragraph 1 ). 
Seed voxels are determined by simultaneously matching 
correspondence to a plurality of pixels in three or more images 
representing a plurality of views of the scene, wherein the base 
image is one of at least three views of the scene, each of the at 
least three views of the scene being situated in a non-linear 
arrangement and are further oriented in a plurality of non-parallel 
planes relative to each other (i.e. ensuring that the cross-correlation 
Uu,v,d between the views satisfies Uu 9 v t dUt1, t1 being a 
predetermined threshold - [Chen, VSMM99] page 31, right column, 
paragraph 1 , lines 3-4). See also example 4 on page 34 with the 6 
rotating views of the teapot. 
(12.b.) Tracing pixels in a virtual piecewise continuous depth surface by 
spatial propagation starting from the provided plurality of seed 
pixels in the base image by using the matching and corresponding 
plurality of pixels in the three or more images to create the virtual 
piecewise continuous depth surface viewed from the base image, 
each successfully traced pixel being associated with a depth in the 
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scene viewed from the base image. The surface tracing was 
treated above with respect to step (1 .a). See also [Chen, VSMM99] 
right column, paragraph 2 and Fig. 4. 
[Chen, VSMM99] presents the volumetric stereo matching method entirely as an 
algorithm and forgoes any mention of possible hardware implementations. However, the 
volumetric stereo matching method is shown to have been executed on a computer 
(e.g. SGI workstation - [Chen, VSMM99] page 34, left column, paragraph 2). In this 
case, item (12.a.) would be implemented in computer-readable instructions, say as a 
"image matching module", and item (12.b.) would be implemented in computer-readable 
instructions, say as a "propagation module". The "image matching module" and 
"propagation module" would presumably be resident (electrically coupled) in some 
memory which is, in turn, coupled to a central processing unit (CPU). In this manner, a 
computer implementation of the a volumetric stereo matching method of [Chen, 
VSMM99] would sufficiently conform to the image processing system of Claim 12. 

The following is in regard to Claim 13. As shown above, [Chen, VSMM99] 
suggests a computer implementation of a volumetric stereo matching method that • 
adequately satisfies the limitations of Claims 12. The disclosed volumetric stereo 
matching method operates on photographic images (e.g. [Chen, VSMM99] Figs. 2 or 8). 
This necessitates some means of capturing images, such as a camera(s) (see [Chen, 
VSMM99] page 34, left column, paragraph 1). In order for the CPU to process the 
captured images, the camera would have to be somehow electrically coupled (e.g. via 
some camera interface) to the CPU. Therefore, the features of the image processing 
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system proposed in Claim 13 are inherent to a computer-based implementation of the 
volumetric stereo matching method of [Chen, VSMM99]. 

The following is in regard to Claim 14. As shown above, [Chen, VSMM99] 
suggests a computer implementation of a volumetric stereo matching method that 
adequately satisfies the limitations of Claims 12. As mentioned above the "image 
matching module" and "propagation module" are resident on a memory. Modern 
memory is implemented as an integrated circuit. All modern CPUs are implemented as 
integrated circuits. In this manner, a computer-based implementation of the a volumetric 
stereo matching method of [Chen, VSMM99] would adequately satisfy the limitations of 
Claim 14. 

The following is in regard to Claims 15-17. Claim 15-17 recite essentially the 
same limitations as claim 5-7, respectively. Therefore, with regard to claim 15-17, 
remarks analogous to those presented above relative to claim 5-7 are respectively 
applicable. 

Rejections Under 35 U.S.C. § 103(a) 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed 
or described as set forth in section 102 of this title, if the differences between the 
subject matter sought to be patented and the prior art are such that the subject 
matter as a whole would have been obvious at the time the invention was made 
to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was 
made. 
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8. Claims 27-28 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over [Chen, VSMM99]. 

The following is in regard to Claim 27. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that adequately satisfies the limitations 
of Claims 24. Although [Chen, VSMM99] does not explicitly mention associating the 
normalized cross-correlation (NCC) with a matching cost, the usage of the NCC in 
[Chen, VSMM99] suggests that the two are closely related and could indeed be used 
interchangeably. As mentioned above, [Chen, VSMM99] uses the NCC, Uu.v.d within 
windows associated with each of the propagating fronts (e.g. windows centered about 
the voxel (u,v,d\ on the surface fronts - [Chen, VSMM99] page 31 , left column, lines 1-3 
and right column, paragraph 2, sentence 1) to determine which of two coincident 
surface fronts to continue to propagate ([Chen, VSMM99] page 31 , right column, last 
paragraph, sentence 5). Furthermore, the NCC is indicative of the "correctness" of a 
match ([Chen, VSMM99], Section 3.2, paragraph 1 , sentence 1 ). All of this is suggests 
the that the NCC provides an adequate measure of the matching cost of the 
propagating surface fronts. Therefore, it would have been obvious to one of ordinary 
skill in the art, at the time of the applicant's claimed invention, to relate the matching 
cost of the propagating surface fronts to the computation of the NCC or designate the 
NCC as the matching cost. In either case, the motivation for doing so would have been 
to utilize the cross-correlative properties of the NCC to evaluate the degree to which 
voxels, comprising the surface fronts, represent correct matches. 
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The following is in regard to Claim 28. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that, when modified in the manner 
suggested above, adequately satisfies the limitations of Claims 26. [Chen, VSMM99] 
suggests rectification of at least one pair of images corresponding to the base view of 
the scene and at least one of the other views (e.g. "reference views"). See [Chen, 
VSMM99] Section 2. Therefore, it would have been obvious to one of ordinary skill in 
the art, at the time of the applicants claimed invention, to further perform rectification of 
at least one pair of images corresponding to the base view of the scene and at least one 
of the other views (e.g. "reference views"). The result of rectification is that "scanlines 
[epipolar lines] are parallel in each image, and the corresponding ones are collinear 
across the images" ([Chen, VSMM99] Section 2, paragraph 1, sentence 2 and last 
paragraph). This is known to have the effect of greatly simplifying the problem of finding 
correspondences across stereo images. 

Claims 2-3 and 19-20 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over [Chen, VSMM99], in view of [Zhang94] (Zhang, Deriche, Faugeras, and Luong, "A 
Robust Technique for Matching Two Uncalibrated Images Through the Recovery of 
Unknown Epipolar Geometry", INRIA 1994). 

The following is in regard to Claim 2. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that sufficiently conforms to the depth 



Application/Control Number: 09/927,558 Page 19 

Art Unit: 2623 

recovery method proposed in Claim 1 . The volumetric stereo matching method of 
[Chen, VSMM99] operates on multiple images (see, for instance [Chen, VSMM99] Fig. 
2 and Fig. 11). That is, the depth recovery according to this method includes the 
following: 

(2.a.) Detecting a plurality of image pixels in a first image corresponding 

to a first view of a scene. 
(2.b.) Detecting a plurality of image pixels in at least a second image 

corresponding to a respective at least a second view of the scene. 
The at least a second image deviates from the first image as a 
result of camera relative motion. This is typical of stereo matching 
algorithms (see also [Chen, VSMM99] page 34, Example 4). 
As noted above, the volumetric stereo matching method of [Chen, VSMM99], like most 
stereo matching methods, involves determining correspondences between a plurality of 
pixels in a first image and a plurality of pixels of at least a second image. In other words, 
the method of [Chen, VSMM99] further involves: 

(2.c.) Determining a first two-view correspondence between the plurality 
of detected image pixels in the first image and a plurality of 
detected image pixels in one of the at least a second image 
resulting in a first potential match set of candidate image pixels 
between the first image and the one of the at least a second image. 
This process involves the evaluation of the cross-correlation 
Uu, v,das discussed above with regard to step (1 .b.). 



Application/Control Number: 09/927,558 Page 20 

Art Unit: 2623 

[Chen, VSMM99] further suggests subsequently computing the projective 
reconstruction from the derived correspondence information in order to reconstruct the 
projective 3D structure of the observed scene ([Chen, VSMM99] Section 4, paragraph 

1)- 

[Chen, VSMM99] does not, however, expressly show or suggest: 
(2.d.) Concatenating a plurality of two-view match sets to form a first 

multiple-view potential match set; and 
(2.e.) Determining a second multiple-view potential match set that is a 

refinement of the first multiple-view potential match set. 
[Zhang94], on the other hand, discloses a stereo matching method. Stereo 
matching across disparate views of a scene is generally a fundamental step in depth 
recovery. The stereo matching method involves at least the following steps (refer to 
[Zhang94] Section 6.3 on pages 1 6-1 9): 

(2.c.) Determining an initial set of correspondences or matches (e.g. a 
first two-view correspondence) between the plurality of detected 
image pixels in the first image and a plurality of detected image 
pixels in one of the at least a second image resulting in a first 
potential match set of candidate image pixels (e.g. candidate 
matches - see, for example, [Zhang94] page 10, Section 5, 
paragraph 1 , lines 1-2) between the first image and the one of the 
at least a second image. 
(2.d.) Removing outliers from the initial set of matches (i.e. determining a 
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"multiple-view correspondence" between the plurality of detected 
image pixels in the first image and the plurality of detected image 
pixels in the at least a second image). See, for example, [Zhang94] 
page 18, line 24 3 , in conjunction with the equations for the weights 
Wj on pages 17-18. In this manner, the set of matches or 
correspondences thus obtained (e.g. the "multiple-view 
correspondences") represents a refinement of the "first two-view 
correspondence", resulting in a "second potential match set" of 
candidate image pixels between the first image and the at least a 
second image. 

(2.e.) The "second potential match set" is based at least in part on a 

computation of reprojection error (e.g. residual n- [Zhang94] page 
17, paragraphs 1-2) for matched pixels that resulted from "the 
difference between the /-th observation and its fitted value' 41 " 
([Zhang94] page 17, paragraph 1, lines 1-2). 
The teachings of [Chen, VSMM99] and [Zhang94] are combinable because they 
are analogous art. Specifically, both [Chen, VSMM99] and [Zhang94] disclose methods 
for stereo matching. Therefore, it would have been obvious to one of ordinary skill in the 
art, at the time of the applicant's claimed invention, to use the methodology of [Zhang94] 
- in particular, steps (2.d.)-(2.e.) above - to refine the initial set of correspondences 

3 Lines are counted sequentially from the top beginning with 1 . An equation is treated as a single line, regardless of its length. 

4 It should be understood that the fitted value in this case is a projection of the matched pixels into some view being evaluated and the 
observation is the respective candidate point correspondences {(mij,ni2i)}. See [Zhang94] page 17, last paragraph. 
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derived according to the method of [Chen, VSMM99]. It would have been obvious to 
one of ordinary skill in the art, at the time of the applicants claimed invention, that, in 
applying this refinement process to method of [Chen, VSMM99], the fitted values 
become the reprojected projective points that delimit the projective structure of the 
observed scene (i.e. "matched pixels that resulted from a projective reconstruction of 
the first potential match set"). The motivation for using the method of [Zhang94], in this 
manner, would have been to detect and remove outliers (e.g. bad locations and/or false 
matches - [Zhang94] page 16, last paragraph) from the initial set of candidate matches 
(e.g. those obtained via step (2.c.) above). See [Zhang94] Section 6.3. 

The following is in regard to Claims 19. Claim 19 recites essentially the same 
limitations as claim 2. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 2). Therefore, with regard to claim 19, remarks analogous to 
those presented above relative to claim 2 are applicable. 

The following is in regard to Claim 3. As shown above the methods of [Chen, 
VSMM99] and [Zhang94] can be combined to satisfy the limitations of Claim 2. 
[Zhang94] further disclose that the "outlier-less" set of correspondences (i.e. the 
"second potential match set") is based at least in part on a least median of squares 
computation (LMedS - [Zhang94] Abstract and page 17, lines 4-7 and paragraph 2) of 
the reprojection errors (i.e. residuals n-see above) related to matched pixels in the first 
potential match set. LMedS, as put forth by [Zhang94], has an advantage over other 
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methods in that it "is very robust to false matches as well as outliers due to bad 
localization" ([Zhang94] page 17, paragraph 1, last sentence). Therefore, it would have 
been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to use the LMedS of [Zhang94] to detect and remove outliers from the first 
potential match set. 

The following is in regard to Claim 20. Claim 20 recites essentially the same 
limitations as claim 3. (The 3D reconstruction instructions merely implement the depth 
recovery method of Claim 3). Therefore, with regard to claim 20, remarks analogous to 
those presented above relative to claim 3 are applicable. 

9. Claims 10-1 1 and are rejected under 35 U.S.C. 103(a) as being 
unpatentable over [Chen, VSMM99], in view of [Okutomi93] (Okutomi and Kanade "A 
Multiple-Baseline Stereo", IEEE 1993), in further view of [Lewis95] (Lewis, "Fast 
Normalized Cross-Correlation", 1995). 

The following is in regard to Claim 10. As shown above, [Chen, VSMM99] 
discloses a volumetric stereo matching method that sufficiently conforms to the depth 
recovery method proposed in Claim 9. As discussed above, the method of [Chen, 
VSMM99] involves the computation of the NCC between a first image window of a pre- 
determined size in the base view, and a second image window of the same pre- 
determined size in other views ([Chen, VSMM99] Section 3.1). It was also show above 
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that the NCC is intimately related to the matching cost. Despite this, [Chen, VSMM99] 
does not expressly show or suggest that the matching cost is determined by computing 
the summation of all the normalized cross-correlations between a first image window of 
a pre-determined size in the base view, and a second image window of the same pre- 
determined size in a reference view. 

[Okutomi93] disclose a stereo matching method that uses multiple stereo pairs 
with various baselines for the purposes of depth recovery ([Okutomi93] Abstract 
- sentence 1). According to [Okutomi93] ( the summation of the sum of squared 

differences (SSD) from multiple stereo pairs can be used to indicate the "correctness" of 
a set of matching points (pixels). See [Okutomi93] page 353 (right column, paragraph 3, 
first and last sentence), page 354, (left column, lines 7-10), page 355 (Section B, 
paragraph 2, sentence 1 ), and equation (39) on page 362. This summation is analogous 
to a matching cost in that it approaches a minimum for correct matches ([Okutomi93] 
page 353, right column, paragraph 3, last sentence). According to equation (39) the 
computation is made within a predetermined window \N belonging to each of the stereo 
images ([Okutomi93] page 354, right column, paragraph 1, line 4). In summary, 
[Okutomi93] shows a "matching cost" determined by computing the summation of all the 
SSD's between a first image window of a pre-determined size in the base view and a 
second image window of the samepre-determined size in other views (e.g. "reference 
views"). Given this, it would have been obvious to one of ordinary skill in the art, at the 
time of the applicant's claimed invention, to use a "matching cost" (i.e. a measure of the 
similarity or "correctness" of a match) determined by computing the summation of all 
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the SSD's between a first image window of a pre-determined size in the base view and 
a second image window of the same pre-determined size in other views (e.g. "reference 
views"). As pointed out by [Okutomi93], this would have the effect of advantageously 
reducing global mismatches ([Okutomi93], page 353, right column, paragraph 3, first 
sentence). 

Note that the SSD and NCC are closely related mathematically. As measures of 
similarity among regions of images, they are functionally equivalent. See [Lewis95] 
Section 2, paragraph 1 . Furthermore, it can also be shown that the process of 
minimizing the SSD and maximizing the NCC are in fact equivalent operations. All of 
this suggests that SSD and NCC are essentially interchangeable. Moreover, the NCC 
has the advantage over the SSD in that it is invariant to "changes in image amplitude 
such as those caused by changing lighting conditions across the image sequence" 
([Lewis95] page 2, lines 8-17). Since lighting conditions are likely to vary across the 
various stereo images, it would have been obvious to one of ordinary skill in the art, at 
the time of the applicant's claimed invention, to substitute the NCC for the SSD in the 
stereo matching method of [Okutomi93]. This would yield a "matching cost" determined 
by computing the summation of all the NCC's between a first image window of a pre- 
determined size in the base view and a second image window of the same pre- 
determined size in other views (e.g. "reference views"). 

The teachings of [Chen, VSMM99] and [Okutomi93] are combinable because 
they are analogous art, namely, the art of stereo matching. The teachings of [Lewis95] 
are applicable to both the methods of [Chen, VSMM99] and [Okutomi93] because all 
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relate generally to correlative approaches for evaluating the similarity of windowed- 
regions belonging to a collection of images of a scene. Combining these teachings as 
suggested above yields a method that conforms sufficiently to that of claim 10. 

The following is in regard to Claim 1 1. As shown above the teachings of [Chen, 
VSMM99], [Okutomi93], and [Lewis95] can be combined so a to yield a method that 
adequately satisfies the limitations of Claim 10. [Chen, VSMM99] suggests rectification 
of at least one pair of images corresponding to the base view of the scene and at least 
one of the other views (e.g. "reference views"). See [Chen, VSMM99] Section 2. 
Therefore, it would have been obvious to one of ordinary skill in the art, at the time of 
the applicant's claimed invention, to further perform rectification of at least one pair of 
images corresponding to the base view of the scene and at least one of the other views 
(e.g. "reference views"). The result of rectification is that "scanlines [epipolar lines] are 
parallel in each image, and the corresponding ones are collinear across the images" 
([Chen, VSMM99] Section 2, paragraph 1, sentence 2 and last paragraph). This is 
known to have the effect of greatly simplifying the problem of finding correspondences 
across stereo images. 

Allowable Subject Matter 



10. Claim 29 is allowed. 
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1 1 . The following is a statement of reasons for the indication of allowable 
subject matter. As shown above, [Chen, VSMM99] discloses: 

(29.a.) Tracing at least one parameter surface (e.g. disparity surface), 

each of the at least one parameter surface traced starting from at 
least one predetermined seed pixel point. 

However, no prior art methods were encountered that involved or even suggested the 

step of: 

(29. b.) Calculating a derivative of function E(g) with respect to parameter g 
by using finite difference to minimize the equation of E{g) put forth 
in Claim 29. 

Conclusion 

12. Applicant's amendment necessitated the new grounds of rejction 
presented in the Office Action. Accordingly, THIS ACTION IS MADE FINAL. See 
MPEP 706.07(a). Applicant is reminded of the extension of time policy as set forth in 
37CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
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extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

1 3. Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Wes Tucker whose telephone number is 571-272- 
7427. The examiner can normally be reached on 9AM-5PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on 571-272-7429. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-21 7-91 97 (toll-free). 



Wes Tucker 
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